Method and apparatus maintaining eye contact in video delivery systems using view morphing

ABSTRACT

A view morphing algorithm is applied to synchronous collections of video images from at least two video imaging devices, and interpolating between the images, creates a composite image view of the local participant. This composite image approximates what might be seen from a point between the video imaging devices, presenting the image to other video session participants.

This application claims the benefit of provisional application Ser. No.60/250,955 filed Nov. 29,2000.

TECHNICAL FIELD

This invention relates to the field of video conferencing and inparticular to methods and systems maintaining the appearance of eyecontact between communicants in a teleconference.

BACKGROUND ART

A primary concern with video teleconferencing systems is the frequentlack of eye contact between participants. In the most commonconfiguration, each participant uses a computer monitor on which animage of the second participant is displayed, while a camera mountedabove the monitor captures the image of the local participant fordisplay on the monitor of the second participant. Since participantsfrequently look at the monitor, either at the image of the secondparticipant or elsewhere on the display, rather than directly at thevideo camera, there is the appearance that the participants are notlooking at one another, resulting in an unsatisfactory user experience.

Many prior art solutions to the eye contact problem have incorporatedhalf-silvered, partially transmissive and partially reflective mirrors,or beamsplitters. These solutions have typically incorporated abeamsplitter placed in front of a computer display at a 45 degree angle.In one typical configuration, a video camera, located behind thebeamsplitter, captures an image of the local participant through thebeamsplitter. The local participant views an image of the secondparticipant on the display as reflected by the beamsplitter.

In devices incorporating a conventional CRT, the resulting device isboth aesthetically bulky and physically cumbersome. Furthermore, incases involving an upward facing display, the display is viewable bothdirectly and as reflected by the beamsplitter, greatly distracting thelocal participant. To alleviate this problem, prior solutions, includingthose described in U.S. Pat. Nos. 5,117,285 and 5,612,734 haveintroduced complicated systems involving polarizers or micro-louvers toobstruct a direct view of the upward facing display by the localparticipant. In all cases, the image of the second participant appearsrecessed within the housing holding the display, beamsplitter, and videocamera. The resulting distant appearance of the second participantgreatly diminishes the sense of intimacy sought duringvideoconferencing.

Another series of prior art attempts to alleviate this problem throughthe use of computational algorithms that manipulate the transmitted orreceived video image. For example, U.S. Pat. No. 5,500,671 describes asystem that addresses the eye contact problem by creating anintermediate three-dimensional model of the participant based on imagescaptured by two imaging devices on either side of the local display.Using this model, the system repositions artificially generated eyes atan appropriate position within the image of the local participanttransmitted to the second participant. The resulting image, withartificially generated eyes and a slight but frequent mismatch betweenthe position of the eyes relative to the head and body of theparticipant, is unnatural in appearance. Furthermore, the creation of anintermediate three-dimensional model is computationally intensive,making it difficult to implement in practice.

U.S. Pat. No. 5,359,362 describes a system “using at each station of avideo conferencing system at least a pair of cameras, neither of whichis on the same optical axis as the local monitor, to obtain athree-dimensional description of the speaker and from this descriptionobtaining for reproduction by the remote monitor at, the listener'sstation a virtual image corresponding to the view along the optical axisof the camera at the speaker's station. The partial 3D description atthe scene can be used to construct an image of the scene from variousdesired viewpoints. The three dimensional description is most simplyobtained by viewing the scene of interest, by a pair of cameras,typically preferably aligned symmetrically on either left and right orabove and below, about the optical axis of the monitor, solving thestereo correspondence problem, and then producing the desired twodimensional description of the virtual image for use by the monitor atthe listener's station.

(The) process of creating the desired two-dimensional description foruse as the virtual image consists of four steps, calibration, stereomatching, reconstruction and interpolation. The calibration converts theview from two tilted cameras into two parallel views important forstereo matching. The stereo matching step matches features, such aspixels, between the two views to obtain a displacement map that providesinformation on the changes needed to be made in one of the observedviews. The reconstruction step constructs the desired virtual view alongthe axis between the two cameras from the displacement map and anobserved view, thereby recovering eye contact. The final step is to fillin by interpolation areas where complete reconstruction is difficultbecause of gaps in the desired virtual view that result from limitationsin the displacement map that was formed.”

Note that U.S. Pat. No. 5,359,362 generates its virtual image bytransforming the image obtained by one of the two physical imagingdevices. The resulting image does not reflect any features of the localparticipant that are occluded from the transformed image.

Still other prior art approaches construct a complete mathematical modelof the local participant and his nearby surroundings. This mathematicalmodel is then transmitted to the second participant, where it isreconstructed in a manner providing eye contact. Clearly, such systemsrequire that both the remote and local communicants own and operate thesame videoconferencing device. This presents a significant obstacle tointroduction and widespread adoption of the device.

Consider the prior art as found in U.S. Pat. No. 5,359,632 again. Often,in such stereo matching systems, prior to beginning real-time videoconferencing image processing, a calibration operation is used to obtaininformation describing the positioning and optical properties of theimaging devices. First a camera projection matrix is determined for eachof the imaging devices. This camera projection matrix characterizes thecorrespondence of a point in three-dimensional space to a point in theprojective plane imaged by the video camera. The matrix determined isdependent on the position and angular alignment of the camera as well asthe radial distortion and zoom factor of the camera lens. One prior artapproach employs test patterns and a camera calibration toolboxdeveloped by Jean-Yves Bouguet at the California Institute ofTechnology. This calibration toolbox draws upon methods described in thepapers entitled “Flexible Camera Calibration by Viewing a Plane fromUnknown Orientations” by Zhang, “A Four-step Camera CalibrationProcedure with Implicit Image Correction” by Heikkilä and Silven, “OnPlane-Based Camera Calibration: A General Algorithm, Singularities,Applications” by Sturm, and “A versatile camera calibration techniquefor high accuracy 3D machine vision metrology using off-the-shelf TVcameras and lenses” by R. Y. Tsa and Maybank.

Following the determination of these camera projection matrices, a twodimensional rectifying transform is determined for each of the pair ofimaging devices. The transformation may be determined based on thepreviously determined camera projection matrices, using an approachdescribed in the paper of Fusiello, Trucco, and Verri entitled“Rectification with unconstrained stereo geometry”. The transformation,when applied to a pair of images obtained from the imaging devices,produces a pair of rectified images. In such a set of images, each pixelin a first video camera image corresponds to a pixel in the second imagelocated along a line at the same vertical location as the pixel in thefirst image.

The prior art also includes calculating a dense correspondence betweenthe two generated camera images. Several algorithms are available fordetermining such a dense correspondence including the method describedin the paper of Georges M. Quenot entitled “The ‘Orthogonal Algorithm’for Optical Flow Detection Using Dynamic Programming”. The Abstractstates “This paper introduces a new and original algorithm for opticalflow detection. It is based on an iterative search for a displacementfield that minimizes the L₁ or L₂ distance between two images. Bothimages are sliced into parallel and overlapping strips. Correspondingstrips are aligned using dynamic programming exactly as 2Drepresentations of speech signal are with the DTW algorithm. Two passesare performed using orthogonal slicing directions. This process isiterated in a pyramidal fashion by reducing the spacing and width of thestrips. This algorithm provides a very high quality matching forcalibrated patterns as well as for human visual sensation. The resultsappears to be at least as good as those obtained with classical opticalflow detection methods.”

What is needed is a method for efficient real-time processing of atleast two spatially offset image sequences to create a virtual imagesequence providing a sense of eye contact, which is of great value in anumber of applications including, but not limited to, videoconferencing. The sense of eye contact should operate effectively acrossthe full range of local participant head positions and gaze directions.It must provide a natural view of the local participant for the secondparticipant. It must be aesthetically pleasing and easily operated by atypical user. What is further needed is apparatus efficientlyinterfacing to a standard video conferencing system and providing theadvantages of such methods of generating virtual image sequences.

SUMMARY OF THE INVENTION

To resolve the identified problems found in the prior art, the presentinvention creates a head-on view of a local participant, therebyenhancing the sense of eye contact provided during any of the following:a video conference session, a video phone session, a session at a videokiosk, and a video training session. Note that video conference sessionsinclude, but are not limited to, sessions presented via one or moreprivate communications channels and sessions presented via one or morebroadcast channels.

A view morphing algorithm is applied to a synchronous collection ofimages from at least two video imaging devices. These images areinterpolated to create interpolation images for each of the videoimaging devices. The interpolated images from at least two of the videoimaging devices are combined to create a composite image of the localparticipant. This composite image approximates a head-on view of thelocal participant providing excellent eye contact.

It should be noted that the synchronous image collection is comprised ofimages received at approximately the same time.

It is often preferred to interpolate the images to a point between thevideo imaging devices when they are placed in a radially symmetricmanner about the local participant. It may be preferred, when the videoimaging devices are not placed in a radially symmetric relationship withthe local participant, that a more complex mechanism potentiallyinvolving partial extrapolation may be used to create what is identifiedherein as the interpolated images.

The video imaging devices are preferably placed on opposite sides of alocal display and the composite image further approximates essentiallywhat might be seen from the center of that local display.

This head-on view of the local participant supports the localparticipant looking directly at the monitor and provides a sense of eyecontact when viewed by the second participant, actively aiding the senseof personal interaction for all participants.

Certain embodiments of the invention include, but are not limited to,various schemes supporting generation of the composite image, control ofcomposite image generation by at least one of the second participants,and adaptively modifying the current images at certain stages based uponremembered displacements from previous images. These embodimentsindividually and collectively aid in improving the perceived quality ofeye contact.

Aspects of the invention include, but are not limited to, devicesimplementing the methods of this invention in at least one of thefollowing forms: dedicated execution engines, with or withoutinstruction processing mechanisms; mechanisms involving table lookup ofvarious non-linear functions; and at least one instruction processingcomputer performing at least some of the steps of the methods as programsteps residing within memory accessibly coupled with the computer.

These and other advantages of the present invention will become apparentupon reading the following detailed descriptions and studying thevarious figures of the drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A shows a simplified block diagram overview of the invention,including local participant 10, video display 30, pair of imagingdevices 41 and 42, means for generating composite image 100, motionvideo portal 70, video delivery system 80 and second participant 90;

FIG. 1B shows a simplified block diagram of an alternative embodiment ofthe invention to FIG. 1A, with motion video portal 70 including firstcomputer 200 with a program system 1000 at least in part generatingcomposite image 146;

FIG. 2 shows a diagram of the preferred positioning of imaging devices41 and 42 relative to local participant 10 as found in FIGS. 1A and 1B;

FIG. 3A depicts a detail flowchart of first program system 1000 of FIG.1B implementing a method of conveying eye contact of a local participantpresented to at least one second participant in a video delivery sessionas a motion video stream based upon observations by an imaging devicecollection;

FIG. 3B depicts a detail flowchart of operation 1022 of FIG. 3A forcalculating the dense correspondence;

FIG. 4A depicts a detail flowchart of operation 1032 of FIG. 3 forgenerating the interpolated image, for each of the pixels of theinterpolated image;

FIG. 4B depicts a detail flowchart of operation 1042 of FIG. 3 forcombining the interpolated images is further comprised, for each of thepixels of the composite image;

FIG. 4C depicts a detail flowchart of operation 1042 of FIG. 3 forcombining corresponding pixels;

FIG. 5A depicts a detail flowchart of operation 1112 of FIG. 4C forcombining corresponding pixels;

FIG. 5B depicts a detail flowchart of operation 1132 of FIG. 5A forpredominantly combining the corresponding pixel of the firstinterpolated image whenever the composite image pixel is a member of thefirst side collection;

FIG. 6A depicts a detail flowchart of operation 1142 of FIG. 5A forpredominantly combining the corresponding pixel of the secondinterpolated image whenever the composite image pixel is a member of thesecond side collection;

FIG. 6B depicts a detail flowchart of operation 1152 of FIG. 5A formixedly combining the corresponding pixels of the at least twointerpolated images whenever the composite image pixel is a member ofthe center collection;

FIG. 7 depicts a detail flowchart of operation 1176 of FIG. 5B forpredominantly combining the corresponding first interpolated imagepixel;

FIG. 8 depicts a detail flowchart of operation 1196 of FIG. 6A forpredominantly combining the corresponding second interpolated imagepixel;

FIG. 9A depicts a detail flowchart of operation 1216 of FIG. 6B formixedly combining the corresponding pixel of the at least twointerpolated images;

FIG. 9B depicts a detail flowchart of operation 1412 of FIG. 9A forcalculating the blending linear combination;

FIG. 10A depicts a detail flowchart of operation 1462 of FIG. 9B forcalculating the bulging scale linear combination;

FIG. 10B depicts a detail flowchart of operation 1012 of FIG. 3 forobtaining the digital version of the image from imaging devicecollection member as the image member in the synchronized imagecollection, for each of the imaging device collection members;

FIG. 11A depicts a detail flowchart of method of operation and programsystem 1000 of FIGS. 1B and 3 for generating the composite image, for atleast two of the imaging device collection members;

FIG. 11B depicts a detail flowchart of operation 1012 of FIGS. 1B and 3for obtaining the digital version of the image, for each of the at leasttwo imaging device collection members;

FIG. 11C depicts a detail flowchart of operation 152 of FIG. 11B forwarping the image digital version;

FIG. 12A depicts a detail flowchart of operation 1572 of FIG. 11C forattenuating the displacement factor for the imaging device collectionmember to modify the displacement factor;

FIG. 12B depicts a detail flowchart of operation 1592 of FIG. 12A formultiplying the displacement factor for the imaging device collectionmember comprised of an operational member of this flowchart;

FIG. 13A depicts a detail flowchart of operational method and/or programsystem 1000 of FIGS. 1B and 3 for generating the composite image;

FIG. 13B depicts various imaging device collection member placements inpotential relationship with display 30;

FIG. 14A depicts a detail flowchart of operational method and programsystem 1000 of FIGS. 1A, 1B and 3 for generating the composite image;

FIG. 14B depicts a detail flowchart of operational method and programsystem 1000 of FIGS. 1A, 1B and 3 for generating the composite image,for at least two of the imaging device collection members;

FIG. 14C depicts a detail flowchart of operational method and programsystem 1000 of FIG. 1A, 1B and 3 for generating the composite image;

FIG. 15A depicts a detail flowchart of operation 1872 of FIG. 14C forspecifying the point P; and

FIG. 15B depicts a detail flowchart of operational method and programsystem 1000 of FIGS. 1A, 1B and 3A for generating the composite image.

DETAILED DESCRIPTION OF THE INVENTION

FIG. 1A shows a simplified block diagram overview of the invention,including local participant 10, video display 30, pair of imagingdevices 41 and 42, means for generating composite image 100, motionvideo portal 70, video delivery system 80 and second participant 90.

Means 100 for generating composite image 146 is communicatively coupled114 and 112 with at least two imaging device collection members 41 and42, respectively. Means 100 regularly receives an image 118 and 116 fromeach of the at least two imaging device collection members 41 and 42,respectively, to provide a synchronized collection of images based uponobservations of at least the local participant's head 10 by the imagingdevices.

Means 100 for generating composite image 146 is communicatively coupled142 to motion video portal 70, providing a succession of compositeimages 146, each based upon at least synchronized image collection 116and 118 to 72 video delivery system 80.

Video delivery system 80 presents 82 second participant 90 motion videostream 72 generated by motion video portal 70 conveying eye contactbased upon the succession of composite images 146. Note that the motionvideo stream is compatible with a digital motion format and/or an analogmotion format. The digital motion format includes, for example, any ofthe following: MPEG1 format, MPEG2 format, MPEG4 format, H.261 formatand H.263 format. The analog format includes, for example, any of thefollowing: NTSC format, PAL format, and SECAM format.

A primary responsibility of video delivery system 80 is to initiate andmaintain a video delivery session with at least one remote location.Note that in various embodiments of the invention, the video deliverysession may include, but is not limited to, any of the following: avideo conference session involving at least local participant 10 and atleast one second participant 80, a video phone session involving localparticipant 10 and second participant 80, a video kiosk supporting videocommunication between at least local participant 10 and at least onesecond participant 80, video training between at least local participant10 and at least one second participant 80, and television broadcastconveying a documentary style interview. Each of these video deliverysessions is based upon the motion video stream presented 72 to the videodelivery system 80 from motion video portal 70.

Video delivery system 80 connects 82 to second participant 90. Theconnection 82 can include transport across at least one communicationsnetwork. While not shown, there is typically another motion video streamfrom second participant 90 which is transported via 82 through videodelivery system 80 and rendered for presentation on video display 30.

Additionally, certain embodiments of the invention may offer an abilityto view the composite image 146 obtained from means 100 on the localvideo display 30. There may further be the ability to view digitalversions of the images 118 and 116 obtained from the video imagingdevices 41 and 42.

A number of existing technologies are suitable for use as video display30 including, for example, cathode ray tube monitors, liquid crystaldisplays, and plasma screen televisions. The display is preferablycompatible with the format of the video output signal provided by thevideo delivery system.

Note that in certain embodiments of the invention, means 100 may be atleast part of an instruction-processing computer and/or a dedicatedhardware accelerator.

Note that as used herein, an instruction-processing computer includes,but is not limited to, single instruction and multiple instructionprocessing mechanisms acting upon single datapaths and multipledatapaths, leading to the often used acronyms of SISD, SIMD, MISD, andMIMD computers.

The instructions processed by instruction processing mechanisms include,but are not limited to, instructions which are directly executed toalter the state of the system they control, as well as instructionswhich alter by inference the state of the system they control. Note thatinstruction execution may be hardwired into the instruction processor,or interpreted. Inferential systems include, but are not limited to,artificial neural networks, logic programming systems, and contentaddressable memory driven control systems.

As used herein, a dedicated hardware accelerator provides at least onemeans by which calculations upon picture entities, preferably at leastpixel components, may be performed. A dedicated hardware accelerator mayor may not include an instruction processing control mechanism.

By way of example, a hardware accelerator may include a state machinecontroller operating at least one partition of its controls as aones-hot state machine. It may be a collection of state machines, withat least one, some or all of these state machines not having aninstruction register. Examples of such state machines often include, butare not limited to, floating point calculators, FIFOs, and bit packingcircuits such as Huffman coders and decoders.

Local participant 10 of the video delivery session is observed by atleast a pair of video imaging devices 41 and 42. The imaging devicecollection members 41 and 42 are collectively disposed to revealessential features, for example, the head of local participant 10 forobservation by at least one of imaging device collection members 41 and42.

Note that each of the digital versions of images 118 and 116 iscomprised of a two-dimensional array of pixels of approximately the samesize and shape. For the sake of discussion, video imaging device 41 isthe first imaging device and video imaging device 42 is the secondimaging device.

Means 100 is comprised of the following:

Means 110 for obtaining a digital version of the image 118 and 116 fromeach of at least two imaging device collection members 41 and 42,respectively, as the image member in the synchronized image collection.

One embodiment of the invention comprise s means 120 for calculating adense correspondence to determine a displacement in at least a firstdimension for each of the pixels in the first image digital version 116to move each of the pixels to a most nearly corresponding pixel in theimage digital versions of at least one other member of the imagingdevice collection 118.

Means 130 for generating an interpolated image 136 and 138, for each ofthe imaging device collection members 41 and 42, respectively. Theinterpolated images 136 and 138 are comprised of a two dimensional arrayof pixels of approximately the same size and shape.

Means 140 for combining at least two of the interpolated images 136 and138 employs a partitioned averaging scheme using at least a seconddimension to create the composite image 146.

Note that the definition of first dimension and second dimension as usedherein is discussed with respect to FIG. 13B.

The pixels may use, for example, any of the known encoding schemes fordesignating at least chrominance and luminance, including but notlimited to, YUV, RGB, and the various CIE derived pixel coding schemessuch as described in the background of the invention. Note that some butnot all embodiments of the invention may require conversion between twoor more encoding schemes.

Conversion between these coding schemes may be performed, for example,by any of the following mechanisms: table look up, numeric calculationand/or compiled logic structures further including but not limited tofinite state machines, logic equations, and truth tables. Note that atable look up of a 24 bit pixel value input generating a 24 bit pixelvalue output requires 48 megabytes of memory.

FIG. 1B shows a simplified block diagram of an alternative embodiment ofthe invention to FIG. 1A, with motion video portal 70 including firstcomputer 200 with a program system 1000 at least in part generatingcomposite image 146.

Program system 1000 is comprised of program steps residing in memory 210accessibly coupled 212 to first computer 200.

Note that the invention includes an apparatus receiving the imagecollection 136 and 138 that may be stored in a memory, such as memory210. The invention may further include various means for obtaining atleast one of images 136 and 138

Note that means 110 for obtaining the digital version from at least oneof the imaging device collection members may include any of thefollowing:

A frame grabbing circuit 220 coupled 112 to imaging device collectionmember 42 for obtaining the image 116 from the imaging device as theimage member in the synchronized image collection 118 and 116.

Video interface 240 coupling 114 imaging device collection member 41 tomotion video portal 70 for obtaining a digital version of image 118 fromimaging device collection member 41.

Obtaining a digital version of an image may also include the step ofperforming a rectifying transformation.

Note that it is preferred with today's technology that a consistentinterface be provided for at least pairs of video imaging devices. It iscontemplated that one of the two alternatives discussed in FIG. 1B wouldbe used for at least pairs of video imaging devices.

The motion video portal 70 may further include any of the following: Afirst finite state machine 230 receiving digital version of image 118from imaging device collection member 41 by operating 232 videointerface 240. A first computer 200 coupled (not shown) with videointerface 240 and accessibly coupled 212 to a first memory 210 andcontrolled by first program system 1000 comprised of at least oneprogram step residing in the first memory 210.

FIG. 2 is a diagram showing the preferred positioning of imaging devices41 and 42 relative to local participant 10, as found in FIGS. 1A and 1B.

Imaging devices 41 and 42 are positioned at a common radial displacementR from the point of intersection C of the video camera field of viewcenterlines. The angular separation of the imaging devices, θ, ispreferably the smallest allowable separation given the size of videodisplay 30 (not shown) and the housing size of imaging devices 41 and42.

Imaging devices 41 and 42, as well as intersection point C of thecenterlines, lie approximately in a horizontal plane. Local participant10 is preferably positioned such that his facial features areapproximately located at C.

Means 100 receives the video signals from imaging devices 41 and 42,respectively, and from these video signals, creates an image of localparticipant 10 as viewed from a point P along the arc common arc A aboutthe point C.

To maximize compatibility with existing video delivery equipment, means100 may receive video input from the imaging devices and provide videooutput to the video delivery system in any one of a variety of videoformats via a variety of transmission protocols. These video formatsinclude but are not limited to analog formats and digital formats. Thedigital formats may include but are not limited to any of bitmap,grayscale, RGB, DV, YUV, and HDTV. The analog formats may include, butare not limited to, any of RS170, RS343, NTSC, PAL1, SECAM, and HDTV.

As used herein, the term digital refers to any communications protocolor format based upon ordered collections of digits. Each digit ispreferably a member of a single digit value collection containingfinitely many digit values. In today's technology, the preferred digitalvalue collection has two members, usually denoted as ‘0’ and ‘1’.

Digital formats are particularly convenient because they allow forsimple conversion of image data into a form easily manipulated by theimage processing algorithm. If a digital format is selected, a transferprotocol, such as USB or IEEE 1394, may be employed. The particularformat of the video output signal is typically selected to match theformat of an existing video camera within the local participant's videodelivery setup, thereby ensuring compatibility with the existing videodelivery system 20.

With the invention configured as described above, local participant 10positions himself or herself relative to local display 30 and imagingdevices 41 and 42 approximately as shown in FIG. 2.

Local participant 10 may check his positioning relative to imagingdevices 41 and 42 by previewing a composite image on local display 30.Local participant 10 may then initiate a video delivery session or joinan existing video delivery session as provided by video delivery system80. After the videoconference, local participant 10 closes the videodelivery session as provided by video delivery system 80.

Prior to beginning the image processing operation, a calibrationoperation is preferably performed to obtain information describing thepositioning and optical properties of the two imaging devices. Thecalibration process may be performed upon assembly of theteleconferencing apparatus if the video camera setup is a permanentfixture, or may be performed each time a change is made to the physicalgeometry or optical settings of the imaging devices.

FIG. 3A depicts a detail flowchart of first program system 1000 of FIG.1B implementing a method of conveying eye contact of a local participantpresented to at least one second participant in a video delivery sessionas a motion video stream based upon observations by an imaging devicecollection.

Arrow 1010 directs the flow of execution from starting operation 1000 tooperation 1012. Operation 1012 performs obtaining a digital version ofthe image from each of the members of the imaging device collection asthe image member in the synchronized image collection. Arrow 1014directs execution from operation 1012 to operation 1016. Operation 1016terminates the operations of this flowchart.

Certain embodiments of the invention include the following operationswithout operation 1012.

Arrow 1020 directs the flow of execution from starting operation 1000 tooperation 1022. Operation 1022 performs calculating at least one densecorrespondence to determine a displacement in at least a first dimensionfor each of the pixels in the first image digital version that wouldmove each of the is pixels to a most nearly corresponding pixel in theimage digital version of at least one other member of the imaging devicecollection. Arrow 1024 directs execution from operation 1022 tooperation 1016. Operation 1016 terminates the operations of thisflowchart.

Arrow 1030 directs the flow of execution from starting operation 1000 tooperation 1032. Operation 1032 performs generating an interpolated imagefor at least two of the imaging device collection members from the atleast one dense correspondence of the at least two images. Arrow 1034directs execution from operation 1032 to operation 1016. Operation 1016terminates the operations of this flowchart.

Each of the interpolated images is comprised of a two-dimensional arrayof pixels of approximately the same size and shape.

Arrow 1040 directs the flow of execution from starting operation 1000 tooperation 1042. Operation 1042 performs combining at least two of theinterpolated images employing, for example, a partitioned or otheraveraging scheme in a second dimension to create the composite imagepresented to a motion video portal creating the motion video stream.Arrow 1044 directs execution from operation 1042 to operation 1016.Operation 1016 terminates the operations of this flowchart.

Note that in various embodiments of the invention none, some or all ofthese steps may be found as program steps residing in first memory 210accessibly coupled 212 to at least one computer 210 contained withinmotion video portal 70.

Note that means 100, 110, 120, 130, and 140 of FIG. 1A may each includeis at least one finite state machine and/or at least one computer. Eachcomputer is accessibly coupled to a memory and controlled by a programsystem made up of program steps implementing the method of operation1000 and individual program steps 1012, 1022, 1032, and 1042,respectively, as shown in FIG. 3A.

Note that multiple computers may access a shared memory accessiblycoupled to each of them.

FIG. 3B depicts a detail flowchart of operation 1022 of FIG. 3A forcalculating the dense correspondence.

Arrow 1060 directs the flow of execution from starting operation 1022 tooperation 1062. Operation 1062 performs calculating a densecorrespondence to determine a displacement in at least a first dimensionfor each of the pixels in the first image digital version which wouldmove each of the pixels to a most nearly corresponding pixel in theimage digital versions of at least one other member of the imagingdevice collection. Arrow 1064 directs execution from operation 1062 tooperation 1066. Operation 1066 terminates the operations of thisflowchart.

FIG. 4A depicts a detail flowchart of operation 1032 of FIG. 3 forgenerating the interpolated image, for each of the pixels of theinterpolated image.

Arrow 1070 directs the flow of execution from starting operation 1032 tooperation 1072. Operation 1072 sets the interpolated image pixel to thecorresponding pixel of the image digital version where the interpolatedimage pixel displaced by a partial displacement in at least a firstdimension for the image device collection member. Arrow 1074 directsexecution from operation 1072 to operation 1076. Operation 1076terminates the operations of this flowchart.

FIG. 4B depicts a detail flowchart of operation 1042 of FIG. 3 forgenerating each of the pixels of the composite image by combining theinterpolated images.

Arrow 1090 directs the flow of execution from starting operation 1042 tooperation 1092. Operation 1092 performs combining corresponding pixelsof each of the interpolated images employing the averaging schemepartitioned along a second dimension to create the pixel of thecomposite image. Arrow 1094 directs execution from operation 1092 tooperation 1096. Operation 1096 terminates the operations of thisflowchart.

Note that the sum of the partial displacements of the image devicecollection members is approximately equal to the displacement. Incertain embodiments of the invention, the partial displacements mustbelong to a limited collection of incremental values, often a range ofintegers. The partial displacements may then sum to an incremental valueclose to the displacement. Suppose the displacement is ‘3’ pixels, withthe first and second partial placements may each be ‘1’. Their sum, as‘2’, is approximately equal to ‘3’.

Various embodiments of the invention may alternatively includedisplacement fractions exactly summing to the displacement. This can beachieved, at least in part, by the use of partial displacementsincluding more than just integers.

It is preferred that each of the pixels of any of the images arepartially ordered in the one dimension by membership in exactly onemember of a partition collection. FIG. 4C depicts a detail flowchart ofoperation 1042 of FIG. 3 for combining corresponding pixels.

Arrow 1110 directs the flow of execution from starting operation 1042 tooperation 1112. Operation 1112 performs combining corresponding pixelsof the interpolated images employing the partitioned averaging schemebased upon the pixel membership in a partition collection to create thepixel of the composite image. Arrow 1114 directs execution fromoperation 1112 to operation 1116. Operation 1116 terminates theoperations of this flowchart.

The partition collection may be comprised of a first side collection ofthe pixels, a center collection of pixels, and a second side collectionof pixels. The center collection is between the first side collectionand the second side collection in the second dimension

FIG. 5A depicts a detail flowchart of operation 1112 of FIG. 4C forcombining corresponding pixels.

Arrow 1130 directs the flow of execution from starting operation 1112 tooperation 1132. Operation 1132 performs predominantly combining thecorresponding pixel of the first interpolated image whenever thecomposite image pixel is a member of the first side collection. Arrow1134 directs execution from operation 1132 to operation 1136. Operation1136 terminates the operations of this flowchart.

Arrow 1140 directs the flow of execution from starting operation 1112 tooperation 1142. Operation 1142 performs predominantly combining thecorresponding pixel of the second interpolated image whenever thecomposite image pixel is a member of the second side collection. Arrow1144 directs execution from operation 1142 to operation 1136. Operation1136 terminates the operations of this flowchart.

Arrow 1150 directs the flow of execution from starting operation 1112 tooperation 1152. Operation 1152 performs mixedly combining thecorresponding pixels of the at least two interpolated images wheneverthe composite image pixel is a member of the center collection. Arrow1154 directs execution from operation 1152 to operation 1136. Operation1136 terminates the operations of this flowchart.

FIG. 5B depicts a detail flowchart of operation 1132 of FIG. 5A forpredominantly combining the corresponding pixel of the firstinterpolated image whenever the composite image pixel is a member of thefirst side collection.

Arrow 1170 directs the flow of execution from starting operation 1132 tooperation 1172. Operation 1172 determines when the composite image pixelis a member of the first side collection. Arrow 1174 directs executionfrom operation 1172 to operation 1176 when the determination is ‘Yes’.Arrow 1188 directs execution to 1180 when the determination is ‘No’.

Operation 1176 performs predominantly combining the corresponding pixelof the first interpolated image to create the composite image pixel.Arrow 1178 directs execution from operation 1176 to operation 1180.Operation 1180 terminates the operations of this flowchart.

FIG. 6A depicts a detail flowchart of operation 1142 of FIG. 5A forpredominantly combining the corresponding pixel of the secondinterpolated image whenever the composite image pixel is a member of thesecond side collection.

Arrow 1190 directs the flow of execution from starting operation 1142 tooperation 1192. Operation 1192 determines when the composite image pixelis a member of the second side collection. Arrow 1194 directs executionfrom operation 1192 to operation 1196 when the determination is ‘Yes’.Arrow 1208 directs execution to 1200 when the determination is ‘No’.

Operation 1196 performs predominantly combining the corresponding pixelof the second interpolated image to create the composite image pixel.Arrow 1198 directs execution from operation 1196 to operation 1200.Operation 1200 terminates the operations of this flowchart.

FIG. 6B depicts a detail flowchart of operation 1152 of FIG. 5A formixedly combining the corresponding pixels of the at least twointerpolated images whenever the composite image pixel is a member ofthe center collection.

Arrow 1210 directs the flow of execution from starting operation 1152 tooperation 1212. Operation 1212 determines when the composite image pixelis a member of the center collection. Arrow 1214 directs execution fromoperation 1212 to operation 1216 when the determination is ‘Yes’. Arrow1228 directs execution to 1220 when the determination is ‘No’.

Operation 1216 performs mixedly combining the corresponding pixels ofthe at least two interpolated images to create the composite imagepixel. Arrow 1218 directs execution from operation 1216 to operation1220. Operation 1220 terminates the operations of this flowchart.

FIG. 7 depicts a detail flowchart of operation 1176 of FIG. 5B forpredominantly combining the corresponding first interpolated imagepixel.

Arrow 1250 directs the flow of execution from starting operation 1176 tooperation 1252. Operation 1252 performs setting the composite imagepixel to include, for example, at least ½ of the corresponding firstinterpolated image pixel. Arrow 1254 directs execution from operation1252 to operation 1256. Operation 1256 terminates the operations of thisflowchart.

Arrow 1260 directs the flow of execution from starting operation 1176 tooperation 1262. Operation 1262 performs setting the composite imagepixel to include, for example, at least ⅞ of the corresponding firstinterpolated image pixel. Arrow 1264 directs execution from operation1262 to operation 1256. Operation 1256 terminates the operations of thisflowchart.

Arrow 1270 directs the flow of execution from starting operation 1176 tooperation 1272. Operation 1272 performs setting the composite imagepixel to include, for example, at least {fraction (15/16)} of thecorresponding first interpolated image pixel. Arrow 1274 directsexecution from operation 1272 to operation 1256. Operation 1256terminates the operations of this flowchart.

Arrow 1280 directs the flow of execution from starting operation 1176 tooperation 1282. Operation 1282 performs setting the composite imagepixel to the corresponding first interpolated image pixel. Arrow 1284directs execution from operation 1282 to operation 1256. Operation 1256terminates the operations of this flowchart.

FIG. 8 depicts a detail flowchart of operation 1196 of FIG. 6A forpredominantly combining the corresponding second interpolated imagepixel.

Arrow 1330 directs the flow of execution from starting operation 1196 tooperation 1332. Operation 1332 performs setting the composite imagepixel to include, for example, at least ¾ of the corresponding secondinterpolated image pixel. Arrow 1334 directs execution from operation1332 to operation 1336. Operation 1336 terminates the operations of thisflowchart.

Arrow 1340 directs the flow of execution from starting operation 1196 tooperation 1342. Operation 1342 performs setting the composite imagepixel to include, for example, at least ⅞ of the corresponding secondinterpolated image pixel. Arrow 1344 directs execution from operation1342 to operation 1336. Operation 1336 terminates the operations of thisflowchart.

Arrow 1350 directs the flow of execution from starting operation 1196 tooperation 1352. Operation 1352 performs setting the composite imagepixel to include, for example, at least {fraction (15/16)} of thecorresponding second interpolated image pixel. Arrow 1354 directsexecution from operation 1352 to operation 1336. Operation 1336terminates the operations of this flowchart.

Arrow 1360 directs the flow of execution from starting operation 1196 tooperation 1362. Operation 1362 performs setting the composite imagepixel to essentially the corresponding second interpolated image pixel.Arrow 1364 directs execution from operation 1362 to operation 1336.Operation 1336 terminates the operations of this flowchart.

FIG. 9A depicts a detail flowchart of operation 1216 of FIG. 6B formixedly combining the corresponding pixel of the at least twointerpolated images.

Arrow 1400 directs the flow of execution from starting operation 1216 tooperation 1402. Operation 1402 performs calculating a fixed linearcombination of the corresponding pixels of the at least two interpolatedimages to create the composite image pixel. Arrow 1404 directs executionfrom operation 1402 to operation 1406. Operation 1406 terminates theoperations of this flowchart.

Arrow 1410 directs the flow of execution from starting operation 1216 tooperation 1412. Operation 1412 performs calculating a blending linearcombination of the corresponding pixels of the at least two interpolatedimages to create the composite image pixel blending in the seconddimension with the composite pixels created by the predominantlycombining steps. Arrow 1414 directs execution from operation 1412 tooperation 1406. Operation 1406 terminates the operations of thisflowchart.

FIG. 9B depicts a detail flowchart of operation 1412 of FIG. 9A forcalculating the blending linear combination.

Arrow 1450 directs the flow of execution from starting operation 1412 tooperation 1452. Operation 1452 performs calculating a sliding scalelinear combination of the corresponding pixels of the at least twointerpolated images to create the composite image pixel blending in thesecond dimension with the composite pixels created by the predominantlycombining steps. Arrow 1454 directs execution from operation 1452 tooperation 1456. Operation 1456 terminates the operations of thisflowchart.

Arrow 1460 directs the flow of execution from starting operation 1412 tooperation 1462. Operation 1462 performs calculating a bulging scalelinear combination of the corresponding pixels of the at least twointerpolated images to create the composite image pixel blending in thesecond dimension with the composite pixels created by the predominantlycombining steps. Arrow 1464 directs execution from operation 1462 tooperation 1456. Operation 1456 terminates the operations of thisflowchart.

FIG. 10A depicts a detail flowchart that shows a central partitioningtechnique that may be used, interalia, operation 1216 of FIG. 6B for formixedly combining the corresponding pixel of the at least twointerpolated images.

Arrow 1470 directs the flow of execution from starting operation tooperation 1472. Operation 1472 performs mixedly combining thecorresponding pixels varied about an occlusion center corresponding to ageometric centroid estimate of the local participant in the compositeimage to create the composite image pixelpixel. Arrow 1474 directsexecution from operation 1472 to operation 1476. Operation 1476terminates the operations of this flowchart.

Arrow 1480 directs the flow of execution from starting operation 1216 tooperation 1482. Operation 1482 performs mixedly combining thecorresponding pixels varied in a linear manner in the second dimensionto create the composite image pixelpixel. Arrow 1484 directs executionfrom operation 1482 to operation 1476. Operation 1476 terminates theoperations of this flowchart.

Arrow 1490 directs the flow of execution from starting operation 1216 tooperation 1492. Operation 1492 performs mixedly combining thecorresponding pixels varied in a piece-wise linear manner in the seconddimension to create the composite image pixelpixel. Arrow 1494 directsexecution from operation 1492 to operation 1476. Operation 1476terminates the operations of this flowchart.

FIG. 10B depicts a detail flowchart of operation 1012 of FIG. 3A forobtaining the digital version of the image from imaging devicecollection member as the image member in the synchronized imagecollection, for each of the imaging device collection members.

Arrow 1510 directs the flow of execution from starting operation 1012 tooperation 1512. Operation 1512 performs applying a rectifyingtransformation associated with the imaging device collection member tothe image from the imaging device collection member to create thedigital version of the image. Arrow 1514 directs execution fromoperation 1512 to operation 1516. Operation 1516 terminates theoperations of this flowchart.

FIG. 11A depicts a detail flowchart of an optional step in connectionwith the method of operation and program system 1000 of FIGS. 1B and 3Afor generating the composite image, for at least two of the imagingdevice collection members.

Arrow 1530 directs the flow of execution from starting operation 1000 tooperation 1532. Operation 1532 performs determining the rectifyingtransformation associated with the imaging device collection member,based upon a raw image from the imaging device collection member. Arrow1534 directs execution from operation 1532 to operation 1536. Operation1536 terminates the operations of this flowchart.

FIG. 11B depicts a detail flowchart of operation 1012 of FIGS. 1B and 3Afor obtaining the digital version of the image, for each of the at leasttwo imaging device collection members.

Arrow 1550 directs the flow of execution from starting operation 1012 tooperation 1552. Operation 1552 performs warping the image digitalversion for the imaging device collection member by the partialdisplacement for the imaging device collection member to modify thedigital version image for the imaging device collection member. Arrow1554 directs execution from operation 1552 to operation 1556. Operation1556 terminates the operations of this flowchart.

Further, warping the digital versions of these images has been shown insimulation experiments by the inventor to minimize the computationaloverhead in the dense correspondence calculation step. Thisadvantageously decreases the computational effort required to create thecomposite image.

Note that certain embodiments of the invention may actively incorporatethe operations of FIGS. 11A and 11B into a single image operation toachieve approximately the same results of successively performing theseoperations.

FIG. 11C depicts a detail flowchart of operation 1552 of FIG. 11B forwarping the image digital version.

Arrow 1570 directs the flow of execution from starting operation 1552 tooperation 1572. Operation 1572 performs applying an attenuating factorto the partial displacement for the imaging device collection member tomodify the partial displacement for the imaging device collectionmember. Arrow 1574 directs execution from operation 1572 to operation1576. Operation 1576 terminates the operations of this flowchart.

FIG. 12A depicts a detail flowchart, for alternative embodiments of theinvention for operation 1572 of FIG. 11C for attenuating the partialdisplacement for the imaging device collection member to modify thepartial displacement.

Arrow 1590 directs the flow of execution from starting operation 1572 tooperation 1592. Operation 1592 performs multiplying the partialdisplacement for the imaging device collection member by an attenuatingfactor and optionally rounding the multiplication to an integral resultto modify the partial displacement. Arrow 1594 directs execution fromoperation 1592 to operation 1596. Operation 1596 terminates theoperations of this flowchart.

Arrow 1600 directs the flow of execution from starting operation 1572 tooperation 1602. Operation 1602 performs replacing the partialdisplacement for the imaging device collection member by a replacementpartial displacement whenever the partial displacement is within adisplacement interval. Arrow 1604 directs execution from operation 1602to operation 1596. Operation 1596 terminates the operations of thisflowchart.

Such operations as 1602 permit replacement of the partial displacementbased upon its inclusion in a range or interval of displacements. If thepartial displacement corresponds to a displacement fraction between{fraction (1/16)} and {fraction (3/16)}, it may be replaced by a partialdisplacement corresponding to a displacement fraction of ⅛, for example.

Arrow 1610 directs the flow of execution from starting operation 1572 tooperation 1612. Operation 1612 performs replacing the partialdisplacement for the imaging device collection member by a table entryreferenced by the partial displacement. Arrow 1614 directs executionfrom operation 1612 to operation 1596. Operation 1596 terminates theoperations of this flowchart.

Note, the attenuating factor may be between 0.0 and 1.1. In certainpreferred embodiments of the invention, the attenuating factor isbetween 0.90 and 1.00.

FIG. 12B depicts a detail flowchart of operation 1592 of FIG. 12A formultiplying the partial displacement for the imaging device collectionmember.

Arrow 1730 directs the flow of execution from starting operation 1592 tooperation 1732. Operation 1732 performs rounding upward the result ofthe partial displacement for the imaging device collection membermultiplied by the attenuating factor to modify the partial displacement.Arrow 1734 directs execution from operation 1732 to operation 1736.Operation 1736 terminates the operations of this flowchart.

Arrow 1740 directs the flow of execution from starting operation 1592 tooperation 1742. Operation 1742 performs rounding downward the result ofthe partial displacement for the imaging device collection membermultiplied by the attenuating factor to modify the partial displacement.Arrow 1744 directs execution from operation 1742 to operation 1736.Operation 1736 terminates the operations of this flowchart.

Arrow 1750 directs the flow of execution from starting operation 1592 tooperation 1752. Operation 1752 performs rounding toward zero the resultof the partial displacement for the imaging device collection membermultiplied by the attenuating factor to modify the partial displacement.Arrow 1754 directs execution from operation 1752 to operation 1736.Operation 1736 terminates the operations of this flowchart.

Arrow 1760 directs the flow of execution from starting operation 1592 tooperation 1762. Operation 1762 performs rounding to nearest the resultof the partial displacement for the imaging device collection membermultiplied by the attenuating factor to modify the partial displacement.Arrow 1764 directs execution from operation 1762 to operation 1736.Operation 1736 terminates the operations of this flowchart.

FIG. 13A depicts a detail flowchart of operational method and/or programsystem 1000 of FIGS. 1A, 1B, and 3A for generating the composite imagewhich receives specific displacement fractions from the secondparticipant and replaces the displacement fractions in use with thespecific displacement fractions

Arrow 1790 directs the flow of execution from starting operation 1000 tooperation 1792. Operation 1792 performs receiving via the video deliverysystem from the second participant a specific displacement fraction forthe imaging device collection member, for the at least two of theimaging device collection members. Arrow 1794 directs execution fromoperation 1792 to operation 1796. Operation 1796 terminates theoperations of this flowchart.

Arrow 1800 directs the flow of execution from starting operation 1000 tooperation 1802. Operation 1802 performs replacing the displacementfraction with the specific displacement fraction for the imaging devicecollection member, for the at least two imaging device collectionmembers. Arrow 1804 directs execution from operation 1802 to operation1796. Operation 1796 terminates the operations of this flowchart.

FIG. 13B depicts various potential imaging device collection memberplacements in relationship with display 30.

Note that at least two imaging device collection members may eachinclude equipment containing a Charge Coupled Device (CCD) array. Theequipment may include more than one CCD array per imaging devicecollection member.

At least one of the imaging device collection members may furtherpreferably embody at least one video camera.

At least two imaging device collection members, 41 and 42, arepreferably horizontally positioned with respect to the head of localparticipant 10, as seen through inspection of FIGS. 1A, 2, and 13B.

At least two imaging device collection members, 43 and 44, may bevertically positioned with respect to the head of local participant 10,as seen through inspection of FIGS. 2 and 13B.

At least two imaging device collection members, 45 and 46, oralternatively 47 and 48, may be diagonally positioned with respect tothe head of local participant 10, as seen through inspection of FIGS. 2and 13B.

At least two imaging device collection members may preferably besymmetrically positioned about a local display as seen by localparticipant 10, as seen through inspection of FIGS. 2 and 13B. By way ofexample, any of the pairs 41 and 42, 43 and 44, 45 and 46, oralternatively 47 and 48 display such symmetry. Additionally, groupingsof more than two imaging device collection members may exhibit symmetry.By way of example, the quadruple 41, 42, 43 and 44, as well as thequadruple 45, 46, 47 and 48 display such symmetry.

Note that as used herein, an imaging device collection may preferablyinclude, but is not limited to, two, three and/or four members.

As used herein the first dimension and the second dimension belong to acollection comprising an essentially vertical dimension 60, anessentially horizontal dimension 62, an essentially diagonal dimension64 and 66 as well as an essentially angular dimension 68. As usedherein, these dimensions 60-66 are preferably aligned with two imagingdevice collection members. The essentially angular dimension 68 maypreferably use the approximate center of the pixel array as the angularcenter. Alternatively, the essentially angular dimension may use theocclusion center corresponding to a geometric centroid estimate of thelocal participant in the composite image.

In certain embodiments of the invention, whenever there are exactly twoimaging device collection members being used, the first dimension andsecond dimension may be the same.

Whenever there are an odd number of imaging device collection members inuse, the second dimension may preferably be the essentially angulardimension.

By way of example, consider an embodiment of the invention using threeimaging devices, 43, 45 and 47. The first dimension, for a givencorrespondence, is typically aligned along a line connecting the twoimaging devices for which the correspondence is calculated. Only onesuch first dimension would be horizontal in a three camera arrangementas shown. One possibility, though, is that the first dimension ishorizontal as defined by the epipolar lines of the rectiied images. Notethat rather than just one center collection, as many as three centercollections as well as three side collections of pixels may bepreferred. Note further that while the composite image is comprised ofessentially the array of pixels as discussed previously, there is alsothe potential of mapping individual pixels by an ordering implicit withthe second dimension.

FIG. 14A depicts a detail flowchart of operational method and programsystem 1000 of FIGS. 11A, 1B and 3A for generating the composite image.

Arrow 1830 directs the flow of execution from starting operation 1000 tooperation 1832. Operation 1832 performs the video delivery systempresenting the local participant the motion video stream conveying eyecontact based upon the composite image succession. Arrow 1834 directsexecution from operation 1832 to operation 1836. Operation 1836terminates the operations of this flowchart.

FIG. 14B depicts a detail flowchart of operational method and programsystem 1000 of FIGS. 1A, 1B and 3 for generating the composite image,for at least two of the imaging device collection members.

Arrow 1850 directs the flow of execution from starting operation 1000 tooperation 1852. Operation 1852 performs providing to the motion videoportal a succession of the images from the imaging device collectionmember for the is video delivery system to present to the localparticipant. Arrow 1854 directs execution from operation 1852 tooperation 1856. Operation 1856 terminates the operations of thisflowchart.

FIG. 14C depicts a detail flowchart of operational method and programsystem 1000 of FIGS. 1A, 1B and 3 for generating the composite image.

Arrow 1870 directs the flow of execution from starting operation 1000 tooperation 1872. Operation 1872 performs specifying a point P from whichthe at least two imaging device collection members are displaced. Arrow1874 directs execution from operation 1872 to operation 1876. Operation1876 terminates the operations of this flowchart.

FIG. 15A depicts a detail flowchart of operation 1872 of FIG. 14C forspecifying the point P.

Arrow 1890 directs the flow of execution from starting operation 1872 tooperation 1892. Operation 1892 performs operating a tactile interfacecontrolled by the participant for specifying the point P. Arrow 1894directs execution from operation 1892 to operation 1896. Operation 1896terminates the operations of this flowchart.

Arrow 1900 directs the flow of execution from starting operation 1872 tooperation 1902. Operation 1902 performs specifying the point P basedupon interactions with the participant. Arrow 1904 directs executionfrom operation 1902 to operation 1896. Operation 1896 terminates theoperations of this flowchart.

Arrow 1910 directs the flow of execution from starting operation 1000 tooperation 1912. Operation 1912 performs specifying the point P basedupon interactions with the second participant reported by the videodelivery system. Arrow 1914 directs execution from operation 1912 tooperation 1916. Operation 1916 terminates the operations of thisflowchart.

Arrow 1920 directs the flow of execution from starting operation 1000 tooperation 1922. Operation 1922 performs specifying the location of theparticipant's eyes within the composite image based upon informationfrom the second participant reported by the video delivery system. Arrow1924 directs execution from operation 1922 to operation 1916. Operation1916 terminates the operations of this flowchart.

Note that as used herein, a tactile interface refers to at least one ofa knob, a slider, a touchpad, a mouse, a trackball, and/or a keyboard.

FIG. 15B depicts a detail flowchart of operational method and programsystem 1000 of FIGS. 1A, 1B, and 3A for generating the composite image.

Arrow 1930 directs the flow of execution from starting operation 1000 tooperation 1932. Operation 1932 performs providing a video conferencebetween at least the local participant and at least the secondparticipant based upon the motion video stream. Arrow 1934 directsexecution from operation 1932 to operation 1936. Operation 1936terminates the operations of this flowchart.

Note that the video conference may be only presented to participants, ormay be presented to an audience including more than just theparticipants. Note further that the motion video stream may include morethan motion video stream versions for different participants, as well asnon-participating audiences. These different versions may providecompatibility with more than one video stream format. By way of example,the non-participating audience may receive an analog video format suchas NTSC or PAL, while the participants receive a digital motion formatsuch as MPEG1 or H.261.

Arrow 1940 directs the flow of execution from starting operation 1000 tooperation 1942. Operation 1942 performs providing a video phone sessionbetween the local participant and the second participant based upon themotion video stream. Arrow 1944 directs execution from operation 1942 tooperation 1936. Operation 1936 terminates the operations of thisflowchart.

Arrow 1950 directs the flow of execution from starting operation 1000 tooperation 1952. Operation 1952 performs providing a video kiosksupporting video communication between at least the local participantand at least the second participant based upon the motion video stream.Arrow 1954 directs execution from operation 1952 to operation 1936.Operation 1936 terminates the operations of this flowchart.

Arrow 1960 directs the flow of execution from starting operation 1000 tooperation 1962. Operation 1962 performs providing a video trainingsession between at least the local participant and at least the secondparticipant based upon the motion video stream. Arrow 1964 directsexecution from operation 1962 to operation 1936. Operation 1936terminates the operations of this flowchart.

Note that in certain preferred embodiments, at least one of theseoperations are supported.

Accordingly, although the invention has been described in detail withreference to particular preferred embodiments, persons possessingordinary skill in the art to which this invention pertains willappreciate that various modifications and enhancements may be madewithout departing from the spirit and scope of the claims that follow.

What is claimed is:
 1. An apparatus for presenting a composite imageconveying eye contact with a conference participant, comprising: acomposite image generator communicatively coupled with at least twomembers of an imaging device collection, for repeatedly receiving animage from each image device in said collection at approximately a sametime to provide a sychromized image collection of said images based uponobservations of said participant by said at least two members of saidimaging device collection; wherein said composite image generator iscommunicatively coupled to a motion video portal for providing asuccession of said composite images based upon at least saidsynchronized image collection for presentation to a video deliverysystem; wherein said video delivery system presents a second participantwith a motion video stream conveying eye contact generated by saidmotion video portal based upon said composite image succession; whereinsaid composite image generator is composite of; means for calculating atleast one dense correspondence to determine a displacement in at least afirst dimension for each pixel in a first image for one of said imagingdevice collection members which would move each of said pixels to a mostnearly corresponding pixel in a second image for at least one other ofsaid imaging device collection members; and means for generating aninterpolated image from each of at least two of said synchronized imagecollection members, wherein said interpolated image is comprised of anarray of pixels.
 2. The apparatus of claim 1, wherein said means forgenerating said interpolated image, for each of said pixels of saidinterpolated image, is further comprised of: means for setting each saidinterpolated image pixel to a corresponding pixel of said image, whereinsaid interpolated image pixel is displaced by a partial displacement inat least said first dimension for said imaging device collection member.3. The apparatus of claim 2, said composite image generator furthercomprising: means for receiving a displacement fraction for said imagingdevice collection member, for each of at least two of said imagingdevice collection members, said displacement fraction indicating saidpartial displacement; and means for displacing corresponding pixelswithin each image by said partial displacement.
 4. The apparatus ofclaim 1, further comprising: means for combining at least two of saidinterpolated images to create said composite image.
 5. The apparatus ofclaim 1, further comprising: means for implementing an averaging schemeto create a composite image.
 6. The apparatus of claim 5, furthercomprising: wherein said scheme is a partitioned averaging scheme. 7.The apparatus of claim 1, said means for calculating further comprising:means for additionally calculating in a second dimension to create saidcomposite image.
 8. The apparatus of claim 7, wherein each of saidpixels of any of said images are partially ordered in at least a seconddimension by membership in exactly one member of a partition collectioncomprising at least two disjoint collections of said pixels; and furthercomprising: means for combining corresponding pixels of said at leasttwo interpolated images employing a partitioned averaging scheme basedupon said pixel membership in a partition collection to create saidcomposite image pixel.
 9. The apparatus of claim 8, wherein saidpartition collection is further comprised of a first side collection ofsaid pixels, a center collection of said pixels, and a second sidecollection of said pixels; wherein said center collection is betweensaid first side collection and said second side collection in said atleast a second dimension.
 10. The apparatus of claim 9, wherein saidmeans for combining corresponding pixels is further comprised of: meansfor predominantly combining said corresponding pixel of said firstinterpolated image, to create said composite image pixel whenever saidcomposite image pixel is a member of said first side collection; meansfor predominantly combining said corresponding pixel of said secondinterpolated image to create said composite image pixel whenever saidcomposite image pixel is a member of said second side collection; andmeans for mixedly combining said corresponding pixels of said at leasttwo interpolated images to create said composite image pixel wheneversaid composite image pixel is a member of said center collection. 11.The apparatus of claim 10, said means for predominantly combining saidcorresponding first interpolated image pixel further comprising: meansfor setting said composite image pixel to additively include a portionof said corresponding first interpolated image pixel; means forsubstantially setting said composite image pixel to said correspondingfirst interpolated image pixel; said means for predominantly combiningsaid corresponding second interpolated image pixel further comprising:means for setting said composite image pixel to additively include aportion of said corresponding second interpolated image pixel; means forsubstantially setting said composite image pixel to said correspondingsecond interpolated image pixel; and said means for mixedly combiningsaid corresponding pixel of said at least two interpolated imagesfurther comprising any of: means for calculating a fixed linearcombination of said corresponding pixels of said at least twointerpolated images to create said composite image pixel; and means forcalculating a blending linear combination of said corresponding pixelsof said at least two interpolated images to create said composite imagepixel blending in said at least a second dimension with said compositepixels created by said predominantly combining means.
 12. The apparatusof claim 11, further comprising: means for calculating a sliding scalelinear combination of said corresponding pixels of said at least twointerpolated images to create said composite image pixel blending insaid at least a second dimension with said composite pixels created bysaid predominantly combining means; and means for calculating a bulgingscale linear combination of said corresponding pixels of said at leasttwo interpolated images to create said composite image pixels blendingin said at least a second dimension with said composite pixels createdby said predominantly combining means.
 13. The apparatus of claim 12,further comprising: wherein said averaging scheme uses weights variedabout an occlusion center.
 14. The apparatus of claim 12, furthercomprising: means for calculating either of said sliding scale linearcombination and said bulging scale linear combination using weights forsaid corresponding pixels varied in a linear manner in said at least asecond dimension to create said composite image pixel blending in saidat least a second dimension with said composite pixels created by saidpredominantly combining means.
 15. The apparatus of claim 12, furthercomprising: means for calculating either of said sliding scale linearcombination and said bulging scale linear combination using weights forsaid corresponding pixels varied in a piece-wise linear manner in saidat least a second dimension to create said composite image pixelblending in said at least a second dimension with said composite pixelscreated by said predominantly combining means.
 16. The apparatus ofclaim 1, wherein for each dense correspondence a sum of said partialdisplacements of said imaging device collection members is approximatelyequal to said displacement.
 17. The apparatus of claim 1, furthercomprising: means for obtaining parallel digital images, one image fromeach of said at least two imaging device collection members; and meansfor applying a rectifying transformation associated with said each of atleast two imaging device collection members to said image from saidimaging device collection member to create a parallel digital image. 18.The apparatus of claim 17, said means for applying said rectifyingtransformation, further comprising: a finite state machine for applyingsaid rectifying transformation associated with said imaging devicecollection member to said image from said imaging device collectionmember to create said parallel digital image.
 19. The apparatus of claim1, further comprising: means for warping said images by applying apartial displacement to modify said images.
 20. The apparatus of claim19, further comprising: means for multiplying said partial displacementby an attenuating factor and optionally rounding said multiplication toan integral result to modify said partial displacement.
 21. Theapparatus of claim 1, said composite image generator further comprising:means for determining a rectifying transformation associated with eachof said imaging device collection members.
 22. Method for conveying eyecontact with a participant in a video conference session comprising thesteps of: obtaining at least two images from each of at least twoimaging devices; wherein each of said images is comprised of atwo-dimensional array of pixels of approximately the same size andshape; calculating a dense correspondence to determine a displacement ina first dimension for each of said pixels in at least one first imagewhich would move each of said pixels therein to a most nearlycorresponding pixel in said image of at least one other of said imagingdevices; and generating an interpolated image for each of at least twoof said imaging devices wherein each of said interpolated images iscomprised of essentially said array of pixels.
 23. The method of claim22, further comprising the step of: combining at least two of saidinterpolated images to create a composite image.
 24. The method of claim23, wherein the step of combining said interpolated images is furthercomprised of the step of: combining corresponding pixels of each of saidat least two interpolated images to create said composite image pixel.25. The method of claim 24, wherein the step combining correspondingpixels is further comprised of the step of: combining correspondingpixels of said at least two interpolated images by employing saidpartitioned averaging scheme to create a composite image pixel.
 26. Themethod of claim 25, further comprising a partition collection, saidpartition collection having a first side collection of said pixels, acenter collection of said pixels, and a second side collection of saidpixels; wherein said center collection is between said first sidecollection and said second side collection in said at least a seconddimension.
 27. The method of claim 26, wherein the step combiningcorresponding pixels is further comprised of the steps of: substantiallycombining said corresponding pixel of said first interpolated image tocreate said composite image pixel whenever said composite image pixel isa member of said first side collection; substantially combining saidcorresponding pixel of said second interpolated image to create saidcomposite image pixel whenever said composite image pixel is a member ofsaid second side collection; and mixedly combining said correspondingpixels of said at least two interpolated images to create said compositeimage pixel whenever said composite image pixel is a member of saidcenter collection.
 28. The method of claim 27, wherein the step ofmixedly combining said corresponding pixel of said at least twointerpolated images is further comprised of any of the steps of:calculating a fixed linear combination of said corresponding pixels ofsaid at least two interpolated images to create said composite imagepixel; and calculating a blending linear combination of saidcorresponding pixels of said at least two interpolated images to createsaid composite image pixel blending in said at least a second dimensionwith said composite pixels created by said predominantlycombining,steps.
 29. The method of claim 28, wherein the step ofcalculating said blending linear combination is further comprised of anyof the steps of: calculating a sliding scale linear combination of saidcorresponding pixels of said at least two interpolated images to createsaid composite image pixel blending in at least a second dimension withsaid composite pixels created by said substantially y combining steps;and calculating a bulging scale linear combination of said correspondingpixels of said at least two interpolated images to create said compositeimage pixel blending in at least a second dimension with said compositepixels created by said substantially combining steps.
 30. The method ofclaim 29, further comprising the step of: calculating said linearcombination using weights for said corresponding pixels varied in alinear manner in said at least a second dimension to create saidcomposite image pixel, and blending in said at least a second dimensionwith said composite pixels created by said substantially combining step.31. The method of claim 29, further comprising the step of: calculatingsaid linear combination using weights for said corresponding pixelsvaried in a piece-wise linear manner in said at least a second dimensionto create said composite image pixel, and blending in said at least asecond dimension with said composite pixels created by saidsubstantially combining step.
 32. The method of claim 22, furthercomprising the step of: employing a partitioned averaging scheme tocreate a composite image.
 33. The method of claim 22, said calculatingstep additionally proceeding in at least a second dimension to create acomposite image.
 34. The method of claim 33, further comprising the stepof: combining said composite pixels using weights for said correspondingpixels varied about an occlusion center corresponding to a geometriccentroid estimate of said participant in said composite image to createsaid composite image pixel.
 35. The method of claim 22, wherein the stepof generating said interpolated image is further comprised of the stepof: displacing the pixels of member of the synchronized image collectionby a partial displacement to form said interpolated image.
 36. Themethod of claim 35, further comprising of the steps of: receiving from asecond participant a displacement fraction for at least one of saidimaging devices, said displacement fraction indicating said partialdisplacement; and displacing each of said pixels in said images by saidpartial displacement to provide a resulting composite image.
 37. Themethod of claim 22, wherein the step of obtaining said at least twodigital images is further comprised of the step of: applying arectifying transformation associated with said imaging devices to saidimage to create parallel digital images.
 38. The method of claim 22,wherein the step obtaining said at least two images is further comprisedof the step of: warping said images by applying a partial displacementto each pixel with said images to modify said images.
 39. The method ofclaim 38, further comprising any of the steps of: multiplying saidpartial displacement by an attenuating factor and, optionally, roundingsaid multiplication to obtain an integer result; replacing said partialdisplacement by a replacement partial displacement whenever said partialdisplacement is within a displacement interval; and replacing saidpartial displacement from a table entry referenced by said partialdisplacement.
 40. The method of claim 22, further comprising of the stepof: determining a rectifying transformation associated with each of saidimaging devices.
 41. A method of conveying eye contact with a localparticipant to at least one second participant, comprising the steps of:calculating at least one dense correspondence to determine adisplacement in at least a first dimension for each pixel in a firstimage produced by a first imaging device which would move each of saidpixels to a most nearly corresponding pixel in an image produced byanother imaging device that is displaced from said first imaging devicerelative to said local participant; generating an interpolated imagecomprising essentially an array of pixels for each of said imagingdevices.
 42. The method of claim 41, further comprising the step of:displacing said image pixels by a partial displacement in at least saidfirst dimension to create said interpolated image.
 43. The method ofclaim 41, further comprising the step of: combining at least two of saidinterpolated images by using an averaging scheme to create saidcomposite image.
 44. The method of claim 41, said calculating step beingperformed in at least a second dimension to create said composite image.45. The method of claim 41, wherein the step of calculating said densecorrespondence is further comprised of the step of: calculating at leastone dense correspondence to determine a displacement in essentially saidfirst dimension for each pixel in a first image captured by a firstimage device which would move each of said pixels to a most nearlycorresponding pixel in an image captured by another imaging device. 46.A method of conveying eye contact with a local participant to at leastone second participant, comprising the steps of: calculating at leastone dense correspondence to determine a displacement in at least a firstdimension for each pixel in a first image produced by a first imagingdevice which would move each of said pixels to a most nearlycorresponding pixel in at least a second image produced by at least asecond imaging device that is displaced from said first imaging devicerelative to said local participant; generating an interpolated imagecomprising essentially an array of pixels from each of said imagingdevices; and adaptively modifying subsequent images, at least in partbased upon previous displacements.
 47. The of the claim 46, wherein saidmodifying step warps said subsequent images by a partial displacement.48. The methods of claim 46, further comprising the step of: saiddisplacement.
 49. The method of claim 46, wherein said modifying step isbased upon a time history of said displacement.
 50. The method of claim46, said calculated step comprises the step of: calculating said densecorrespondence between previously calculated dense correspondences.